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ABSTRACT 



This collection of papers presented at a 1996 conference on 
children's mental health focuses on program evaluation efforts within states 
and systems of care. Individual papers have the following titles and authors: 
(1) "An Overview of State Public Mental Health Programs for Children with a 
Serious Emotional Disturbance" (Pamela C. Roddy) ; (2) "A Developmental 

Analysis of Neighborhood Governance: An Aspect of the Annie E. Casey Mental 
Health Initiative for Urban Children's Evaluation" (Marvela Gutierrez -Mayka 
and others); (3) "Promoting Evaluation of Systems of Care: Stakeholder Needs 
and Technical Assistance Strategies" (Susan Flint and others) ; (4) "The 

Comprehensive Services Act: First Year Implementation Assessment" (J. Randy 
Koch and others); (5) "Access Vermont: A Statewide Evaluation of 
Non-categorical Service Delivery to Children and Families- -Meeting Community 
Needs, Measuring Outcomes" (Jennifer Taub and others) ; (6) "Conceptualization 

and Initial Development of the Ohio Scales" (Benjamin M. Ogles and D. Kevin 
Gillespie) ; (7) "Impact of the Child and Adolescent Service System Program in 
California" (June Madsen Clause and others) ; (8) "Evaluation of the 

California Implementation of the Child and Adolescent Service System Program: 
A Review of Documents" (Karyn L. Dresser and others); (9) "Issues in 
Implementing a Randomly Assigned Study Design in a Community Setting: Lessons 
from the Vanderbilt/Stark County Evaluation Project" (Jon Thomas and others) ; 
and (10) "Evolutional Changes in a Local System of Care for Severely 
Emotionally Disturbed Children and Adolescents: The Pendulum Is Swinging 
(Stanley Mong and others). (Individual papers contain references.) (DB) 
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Introduction 

Over the past decade, increasing concern has been expressed about the mental health of our children. 
This has been caused by a number of factors including widespread family breakdown, the growing 
number of children bom to teen age mothers, the increasing rates of substance abuse and school 
dropouts. Many of these children receive no care and of those that do, many receive it from state and 
local public mental health agencies. Thus, it is important to examine how well the states are addressing 
this need. 
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Methods 

The data for this report come from the individual State Federal Block Grant Plan and Implementation 
Reports. The Federal State Block Grant legislation. Public Law (P.L.) 102-321, requires that the states 
develop overall program plans and annual progress reports. The plans must address twelve criteria 
pertaining to the development of community based systems of care. Not surprisingly, the quality of the 
plans and reports varies significantly across the states as does the amount of information provided. 
However, these reports have improved significantly due to better management information systems, 
better record keeping, and the participation of State Planning Councils with consumer and family 
representation. 
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Results 



Data from the State Federal Block Grant Plan and Implementation Reports suggest that all states have 
made progress in establishing an organized community based system of care for children and 
adolescents with serious emotional disturbances (SED) and their families. All 50 states report that they 
have adopted the Federal Child and Adolescent Service System (CASSP) principles for what constitutes 
an organized comprehensive community based system of care. These principles state that the systems 
should be comprehensive, coordinated, community based, family centered and culturally competent. 

In some states, this system development has been further enhanced by initiatives from the Governor, 
state legislators and local government officials. In other states, system development is impacted by 
litigation, especially in regard to inappropriate hospitalization. In still other states, state laws have been 
enacted which mandate that systems or system components such as local interagency coordinating 
councils be developed. 



Furthermore, all the states report efforts to develop interagency collaborations, usually with state and 
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local education, health, child welfare and juvenile justice agencies. Children's mental health service 
agencies are now linked with child welfare and social service agencies in 48 states, with educational 
agencies in 44 states, with juvenile justice agencies in 28 states and with health agencies in 28 states. 

Family Support 

There has been significant growth in family support and family preservation activities. In 1993, 

Congress passed legislation that, for the first time, made a significant investment in preventive services 
aimed at keeping families together. This legislation, known as the Family Support and Preservation Act, 
passed as part of the Omnibus Budget Reconciliation Act (OBRA) of 1993 (P.L. 103-66). As such, it 
was the first major new child welfare legislation in over a decade. In 1 994, family preservation home 
builder programs were established in 35 states, and statutes addressing family preservation in 21 states. 

Concomitant with this, there has been tremendous growth in the movement to establish grass routes 
family support groups throughout the country. 

Serving Children with SED 

States reported serving an increased number of children with SED. The "most in need" target population 
included those persons under 18 years old who had a mental illness; that is, had a DSM-III R diagnosis, 
experienced serious functional impairment, had multi agency involvement, and whose illness duration 
has been greater than one year. Estimates of the prevalence of SED among children vary from 2-5% to 
12% of the child population. While there are varying estimates of need and demand for services, for the 
purpose of this report the most conservative estimates will be used; that is that 2% of all children under 
age 1 8 have a SED, and that 1 .2% of all children under 1 8 are estimated to need services for SED from 
the public sector at any given point in time. 

In an attempt to assess state programs in regard to how many children with SED they serve, these 
numbers were examined on a state-by-state basis. In examining these data, it was clear that there was 
great variation across the states in the numbers of children with SED served in comparison to the 
conservative estimates of target populations. The reports of percent of children with SED served ranged 
from 8 to 100%. Moreover, 14 states exceeded the proportion of children estimated to need services in 
the public sector in FY 1993-1994 (see Table 1). Another six states came close to these targets. These 
states included Alabama, Georgia, Indiana, Kentucky, Maine and South Dakota. 

Development of Community Based Services 

States also reported progress in developing community-based service arrays to substitute for inpatient 
care. According to the Block Grant Reports, the community based services for which there was the 
greatest demand and need were respite, crisis care and home-based services, as well as flexible 
wraparound services. Moreover, in order to further improve access, more and more states reported 
initiatives to provide services in schools and provide single points of service system entry. 

The major financing sources reported were private insurance and/or foundation money, Medicaid, and 
funds from state and local jurisdictions. At a national level, private sources funded about 44% of care, 
with Medicaid funding 19% and state and local jurisdictions funding 28%. During the 1980's Medicaid 
expenditures grew faster than private insurance expenditures. 

Medicaid eligibility procedures appear to have benefited state mental health programs in many ways. 
Requirements for pre-admission screening for hospitals may have prevented inappropriate admissions. 
Additionally, most states reported the ability to obtain Medicaid reimbursement for case management 
and rehabilitation services. Furthermore, many states have or are in the process of obtaining Medicaid 
waivers, despite the outcome of the national health care reform legislation. 

Use of Inpatient Hospital Services 

States report varying degrees of success in reducing use of hospital services, with admission rates 
ranging from 0 per 100,000 children, to over 100 per 100,000 children. The states with the highest rates 
and the ones with the lowest rates are displayed in Table 2. While some states reported success in 
reducing their hospital use, others have not been successful, and still others had mixed results. The states 
which report success number almost 25, and attribute the decrease in use to closing state hospitals and/or 
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reducing the number of beds for children, and developing alternative sources of care. However, seven 
states have experienced increasing rather than decreasing use of hospital care. Connecticut, Idaho, 
Louisiana and Tennessee, report mixed results. For example, in Idaho, the number of admissions 
declined, but total number of bed days and average length of stay increased. Other mixed results 
included those states in which the past declines in use had leveled off and/or reversed direction over the 
past year. These data are presented in Table 3. 

To explain these varying degrees of success, the availability of alternative community services was 
examined, including the reported number of case managers and rate of screening for hospital admission, 
along with the number and array of alternative facilities, such as residential treatment centers (RTCs) 
and therapeutic foster and group homes. In California, for example, RTCs and groups homes were often 
used in lieu of hospitals. 

Case Management 

Reports indicated that Case Management was present in all states but Wyoming, which was in the 
process of developing this service. The percent of children with SED who received case management 
services ranged from about 8% to 100%. As can be seen in Table 4, 15% or fewer such children received 
case management in 1 1 states, whereas 50% or more of the children with SED received case 
management services in 10 other states. When this is compared to the conservative 1.2% estimate of 
children who need to use publicly funded services for SED, there appeared to be somewhat of an inverse 
relation between the states who reported the highest percent of children with SED served, and the 
proportion of those who receive case management. It appeared that the higher the proportion of children 
with SED who receive services, the lower the proportion who receive case management (Table 5). Of 
course, these data do not take into account differing case management models and client ratios. 

Programs for Homeless Children 

Programs for Homeless Children with SED were examined across the 50 states using block grant data 
along with data from the Program to Assist in the Transition from Homelessness (PATH). While all or 
almost all of the states report having programs to address the needs of homeless families and children, 
not all of them addressed homeless children with SED per se, and some did not specify numbers of such 
children served. Instead reference was made to services for homeless families and runaway youth in 
general. These data, however, did indicate that Arkansas, Maine, and North Dakota spent a sizable 
portion of their PATH money on children with SED. Other states such as Georgia, Kentucky and Oregon 
had conducted or were in the process of conducting surveys to assess the number of homeless children 
with SED. In addition, both Kentucky and Washington have received research demonstration grants 
targeted to providing services for homeless children with SED. 
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Introduction 

The Annie E. Casey Foundation's Mental Health Initiative for Urban Children is a five year, 
neighborhood scale program that seeks to demonstrate new ways of delivering comprehensive and 
coordinated, culturally appropriate, and family-sensitive mental health services to low income urban 
communities, and to work with states to improve the policies and practices supporting these services. 
Four urban neighborhoods in the cities of Richmond, Miami, Houston and Boston were selected to 
implement this initiative. 

The Florida Mental Health Institute was contracted to conduct the evaluation of the Initiative's outcomes 
in four areas: (1) overall quality of life in the neighborhood, (2) neighborhood governance, (3) interface 
between family needs and system response, and (4) documentation of movement of children through the 
system and the budget allocations associated with these moves. 

The emphasis on neighborhood governance is one of the unique aspects of the Mental Health Initiative 
for Urban Children which describes this as a mechanism for the delegation of real authority and 
decision-making from government to the neighborhood groups. This paper focuses on the outcomes of 
the process of neighborhood governance development including a description of the method of data 
collection, the creation of a developmental framework for data analysis, and a brief discussion of the 
findings. 
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Methodology 



Data collection for the neighborhood governance study involved two methods: a survey to establish the 
membership, structure, and composition of the neighborhood boards; and open-ended interviews with at 
least two respondents from each of the following three categories: (1) board members/neighborhood 
residents; (2) local level personnel from organizations, agencies or community associations; and (3) 
State agency representatives. 



Once the data were collected, a process involving review of the literature on community organizations, 
feedback from informal contacts with key informants at each site and formal reports from technical 
assistance staff working with the sites, and preliminary review of interview data 
led to the creation of a framework to describe governance board development. This framework depicts 
this development as moving along several attributes to effective administration and planning. Seven 
attributes were identified: (1) Community Information, (2) Community Involvement, (3) 
Consciousness/Knowledge, (4) Linkages. (5) Organization, (6) Programmatic Involvement, and (7) 
Resource Management Capacity (see attached defintitions). 
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Each of the attributes is divided into two or more aspects which define the attribute in more detail. For 
example, the attribute Consciousness/Knowledge is divided into four aspects: Belief in the Capacity to 
Change, Overall Vision of the Initiative, Knowledge of Human Service Systems, and Awareness of 
Political Process. The develop-mental stages for each aspect of an attribute include a continuum of nine 
stages. Stages 1, 3, 5, 7, and 9 have been defined. The intermediate stages 2, 4, 6, and 8 are not defined, 
but are included to allow for variance in the assignment of stages of development. 
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Discussion of Findings 

The analysis has revealed significant similarities across the four sites. The overall development of 
neighborhood governance in three of the sites that have established Boards falls between Stages 1 and 3 
across all developmental attributes. The East Little Havana site had not selected a Board at the time of 
the data collection for this report, and therefore, no developmental stages were assigned. The attributes 
of Consciousness/Knowledge, Organization, Community Involvement and Linkages have been selected 
for discussion in this paper. 

Consciousness/Knowledge 

A common element at all sites was the mixture of optimism and pessimism regarding the Board's 
capacity to influence the process of change in the system. Respondents at the state and local levels show 
varying degrees of optimism about the future of the Initiative. While they recognize the challenges that 
lie ahead, they also emphasize the accomplishments of having brought a diverse group of individuals 
together around this effort. 

At the neighborhood level, however, the resident representatives on the Board are not yet confident 
about their chances to effect meaningful and lasting changes in service systems. Their pessimism seems 
to be rooted in historical factors affecting the relationships between the state, city/county, and the 
community. In particular, residents expressed their disappointment at other initiatives that were 
implemented in the past that left the community untouched and their problems unsolved. A fear that this 
experience may be repeated once more, combined with a sense of frustration over the lack of concrete 
services and programs at this point of the Initiative, are major challenges to the development of trust, a 
sense of partnership, and improved optimism among resident representatives on the Boards. 

Another interesting trend within this attribute is the presence of several visions of the Initiative which 
vary depending on whether the source is the neighborhood, the local level (e.g., city, county providers), 
or the state level. The neighborhood's vision seems to focus on short-term expectations 
in the area of service delivery and program development. Neighborhood residents see this Initiative as an 
opportunity to bring much needed services to the community, and upgrade existing programs to better 
serve children and families. 



The state level, on the other hand, envisions the Initiative as an opportunity to accomplish long-term 
system reform in the area of children's mental health. This level sees programmatic development more as 
a means to an end, rather than as the result of reform itself. 



The local level seems to combine a short and long-term vision of the Initiative. Although, like the state, 
it recognizes system reform as the ultimate goal, like the neighborhood it sees the Initiative as an 
opportunity to supplement existing pools of money that are already being invested in the targeted 
communities (e.g., empowerment zones, Medicaid, full service schools, etc.). Another aspect of the local 
level vision is the perception of the Initiative as a chance to test the viability of new approaches to 
service delivery at the neighborhood level. 



Organization 

The most striking trend with regard to Board organization is the high degree of confusion across all sites 
regarding the roles of the different levels involved in the Initiative. Although in principle all parties 
agree that this is a community-driven effort, and that the neighborhood should be a key player and 
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driving force, in practice, there appear to be conflicting views of the degree to which the neighborhood 
should be in control. 

Residents perceive that their involvement should be at all levels of the decision-making process. They 
also express a need to catch up with the knowledge and expertise of the government level 
representatives, and feel frustrated when they are not treated as equal partners. Local and state level 
officials, on the other hand, are concerned that the community does not have the necessary expertise and 
experience to control the entire process. 

An additional trend identified in the area of Organization relates to the distribution of labor. As a result 
of neighborhood representatives’ need to feel included at all levels of the Initiative, the demands of 
serving on the Board become overwhelming. Tasks are highly concentrated on a few dedicated, 
politically aware volunteers who self-select themselves to be on the Board. These volunteers participate 
on several committees at a time, in addition to attending training events and Board meetings. 

A phenomenon related to the concentration of labor on a few experienced individuals with a history of 
community involvement is the influence of strong personalities on the Board and among the Initiative's 
staff. During the initial phases of planning and implementation, all sites appear to have experienced this 
phenomenon to some extent, and dealt with it differently. In some cases, the identity of the Board came 
to be strongly tied to these individuals, leading other members to feel excluded from the process. 

Community Involvement 

Board representativeness is one of the biggest challenges facing all four Boards. The answer to the 
question of who are the Board's constituents, and therefore, to whom the Board is accountable remains 
elusive. Issues of representativeness cut across several lines including ethnicity, gender, age, 
immigration status, official status, provider status, parental status (e.g., parents of at-risk children), etc. 

The data analyzed for this report suggest that some of the difficulties experienced by the Boards with 
regard to representativeness may be related to the type of base upon which the Boards were founded. 
Boards that used pre-existing organizations reportedly inherited their perceived shortcomings. In Boston, 
the Healthy Boston Coalition's perceived inadequate inclusion of Latinos, other non- African American 
minorities, and parents of at-risk children targeted by the Initiative, is viewed by some as the cause of 
their struggle with representativeness. In Houston, concerns are related to the perceived over 
representation of providers on the Board. Representation challenges of Boards which started from 
scratch, such as Richmond and Miami, are linked to the need to broaden the existing resident 
representation on the Board. 

Linkages 

The main finding with regard to the linkages that the different levels of the Initiative have established 
among themselves is that sites appear to rely on a combination of direct and indirect links. Direct links 
are characterized by the state, local, provider and neighborhood levels communicating directly with each 
other in dyads (e.g., state-to-local, state-to-neighborhood, local-to-neighborhood, etc.). The data suggest 
that this situation has led to the third party feeling excluded from the communication loop. A variation 
of direct links is the inclusion of state and local level representatives on the Board, as in the case of 
Boston and Miami. Indirect links refer to the use of intermediate channels of communication. This could 
be staff persons such as the State Coordinators and Project Director/Managers in Boston and Richmond, 
or an organization such as the Casey Policy Council in Houston, or the Children's Service Council in 
Miami. 

Of particular interest is the response elicited by the participation of providers in this Initiative. Except 
for the Houston Board which drew its original membership from a group of provider organizations, the 
other sites appear to have mixed feelings about any links with providers. These feelings are attributed by 
some to a fear that the Initiative would become provider-driven rather than neighborhood-driven. 

Return to Top 



Closing Remarks 



erIc 



14 



12/10/98 11:26 AM 



8th.9 



http://lumpy.fmhi.usf.edU/CFSroot/rtc/proceeding8th/8th.9.html 



The Developmental Framework for neighborhood governance is a significant first step toward 
describing the mechanisms and events that lead to effective community participation within 
the context of systems reform. The framework's delineation of a tentative path toward neighborhood 
governance should provide a sounding board against which to challenge the proposed course and 
generate new ones. It is hoped that the repeated application of this framework to data generated during 
the remaining years of the neighborhood governance study will further the understanding of the complex 
process designed for this intervention. 
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Introduction 

With the fundamental reconceptualization of mental health care for children and their families (Knitzer, 
1993), and accelerating changes in service financing and delivery, there is increasing need for 
accountability and greater understanding of the impact of these systems changes (Saxe, Richmond, 
Gardner & Cross, 1994). Public sector agencies responsible for mental health services for children are 
facing increased expectations for evaluation, but often lack the resources, tools and experience. 

This paper summary describes a needs assessment conducted by the Technical Assistance (TA Center) 
for the Evaluation of Children's Mental Health Systems that was designed to provide information about 
the evaluation interests and technical assistance needs of three key stakeholder groups. The TA Center 
works with state and local communities to enhance their capacity to conduct and utilize evaluation of 
children's mental health system changes. The results of the needs assessment and a pilot site have guided 
the TA Center's approach to technical assistance strategies and activities. 
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Method 



Three stakeholder groups were included: (1) family members of children with serious emotional 
disorders, (2) state administrators of children's mental health programs, and (3) members of state 
legislatures and their staff. Information was gathered between April and September 1994. 

Family members contributed information through a focus group conducted at the conference held by the 
Research and Training Center on Family Support and Children's Mental Health in Portland, OR in April 
1994. Nine parents, identified through family support/advocacy organizations in their states, 
participated. The purpose of the focus group was to elicit from parents the crucial questions they have 
about service effectiveness and esponsiveness. Participants were also asked about how technical 
assistance might be used to increase families' involvement in evaluation. 

Twenty-three state government children's mental health administrators were interviewed between May 
and September 1994. The sample was drawn from the membership of the State Mental Health 
Representatives for Children and Youth (SMHRCY). Interviews were semi-structured, took between 45 
and 75 minutes to complete, and included queries about system changes, evaluation experience and 
resources, and technical assistance needs related to evaluation. 

State legislators were assembled for a focus group designed to explore their needs for information. The 
forum, held in Washington, DC in June 1994, was attended by ten legislators, legislative staff members 
and other key policy and decision makers. The group addressed several questions, including: what 
information is needed by legislators, at what points in time; what information gaps exist; and in what 
format is information most useful? 



The TA Center provided on-site technical assistance at a community service program, which, in addition 
to providing immediate help, was designed to elicit additional information about 
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the specific needs of customers, the most useful approaches to providing technical assistance, 
and the amount of time and follow-up required. 

Return to Top 



Results 

Family Members 

Information from family members shows that their knowledge of and involvement 
with evaluation activities varied markedly. Family involvement was typically related to satisfaction 
surveys or monitoring for federal block grant requirements. One major barrier to family involvement 
was lack of knowledge about evaluation in general, and absence of information about specific 
evaluations in their states or communities. 

Families hoped for evaluations in the areas of cultural responsiveness of services, service fit (in addition 
to consumer/family satisfaction), and tracking to assess whether those seeking services actually receive 
them. When they are involved at the design stage, families believe they can make important 
contributions clarifying the meaning of inquiries and the language used. Families reported that they need 
concrete examples of the role and importance of program and systems evaluation. 

Administrators of State Child Mental Health Programs 

All states were experiencing one or more major systems change, generally in multiple domains 
simultaneously. The developmental stage of changes varied markedly within and among states. 
Evaluation of systems change had been limited in all but a few states in the sample. The most common 
evaluation activities had been surveys of consumer/family satisfaction and level of involvement. 
Administrators reported that they desire evaluation to understand the progress, impact and effectiveness 
of changes as they are implemented in their states. 

However, while public children's mental health agencies have the responsibility to conduct evaluations, 
they need more technical expertise and resources to carry this out in a meaningful way. Although many 
states had in-house evaluation capacity, university affiliations, or both, such resources were often 
adult-oriented and of limited use for children's mental health systems evaluations. Primary areas of need 
for technical assistance for this group of stakeholders are illustrated in Table 1 . 

Legislators 

A number of key issues were identified by legislators participating in the focus group. They believed 
that too little is known about program outcomes; the crucial question "How does this service effect 
someone's life?" is answered too rarely. Legislators were also aware that basic data are often not 
available or not accurate. They acknowledged that even data about who is being served in which 
programs frequently are inconsistent or incomplete. 

Legislators wanted to know the individual successes or unhappy stories associated with services. They 
suggested that this information helps them understand programs better and helps them interpret 
information to their colleagues. They also wanted to know if a service is cost effective. A multiple 
system perspective is important to legislators because they must make decisions related to a number of 
child-serving agencies all of which have an impact on the others. They indicated that information about 
how changes in one systems effects other systems is helpful. 

Legislators reported a need for information at critical points in the legislative decision-making cycle. 
They need information that is concise and that is customized for the issue or discussion at hand. 

Community Site 

Our understanding of technical assistance needs was furthered through a pilot intervention with Project 
Connect, a local program in the Metro South Region of Massachusetts. This program integrates the 
efforts of agencies and families to develop and implement treatment plans for children with emotional 
disturbances. Through providing consultation on Project Connect's evaluation, as well as reviewing 
documents and interviewing staff, we learned more about the technical assistance needs of an evaluation 
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1 . Time pressures. Needing data quickly for program improvement and funding agency 
requirements. Project Connect rapidly conducted an initial evaluation using focus groups and 
satisfaction surveys with parents, service providers, and referring professionals. We learned that 
longer-term planning for evaluation must be built around the need for quick turnaround on data. 

2. Limited resources for evaluation. Project Connect's limited resources made even modest 
evaluation difficult to implement. In addition to TA Center help. Project Connect recruited 
community volunteers to administer the surveys and graduate students to analyze data. We learned 
that agencies may need assistance to develop resources for basic evaluation functions. 

3. 

Small increments. Project Connect's modest efforts demonstrated the value of developing 
evaluation in small increments. The scope was feasible and widely acceptable, and the results 
were still useful. Family feedback, for example, which suggested overall satisfaction with the 
program, led to program improvements (e.g., more involvement of extended family members), 
and stimulated thinking about what child and family outcomes were reasonable to expect. This 
success encouraged additional evaluation efforts. We learned to support small improvements in 
evaluation that take into account the time pressures and resource limitations faced by children's 
mental health systems and program. 
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Implications 

All representatives of stakeholder groups reported that they want data and outcome information to 
inform policy decisions and program development. They suggested that the most useful data are timely, 
concise, and presented in user-friendly language. They agreed that evaluation findings should be 
disseminated among policy and decision makers, service providers, consumers and families as part of a 
continuous feedback loop. Evaluation information can then be used to improve or change programs, 
make funding decisions, and advocate for new services. 

Stakeholders also share a concern about selecting and measuring outcomes. States need assistance to 
determine which outcomes are appropriate for a state/community, select outcome measures, and gather 
the necessary data. Results of focus groups indicated that outcome information is desired across a 
number of domains, including child and family functioning, consumer/family satisfaction, service fit, 
cost effectiveness, and cross-system collaboration. Many stakeholders were concerned that existing 
instruments are too narrow and too deficit-based. 



There was agreement that children's mental health data are problematic. Data that do exist are embedded 
in the records of categorical agencies and do not give a holistic picture of the child and family. 
Computerized data systems need to be strengthened, both within mental health agencies and across 
agencies. 

Family-professional collaboration in evaluation has not been explored fully. We feel that, as key 
stakeholders, families must be brought more fully into the evaluation process at all points. Family 
members could benefit from information and training about how evaluation can be useful to them and 
how to become meaningful partners with other stakeholders around evaluation. Another issue that is 
central to the philosophy of systems of care is cultural competence, yet assessment of cultural 
competence in children's mental health is at a nascent developmental stage. The TA Center promotes 
evaluation as a collaborative effort, encouraging key stakeholder groups to participate in a range of 
evaluation activities. We believe that evaluations that are fully participatory will reflect and respect the 
participants' cultural, social, and ethnic backgrounds. 



Findings from the needs assessment demon-strate the importance of providing a range of technical 
assistance activities that both demystify and promote evaluation to all stakeholders in children's mental 
health. The TA Center is fostering the use of evaluation to produce useful and timely information that 
will constructively inform and improve systems change. In many places the most useful evaluations may 
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be those that will provide information about collaboration and other structural changes, about 
consumer/family needs and satisfaction, and about intermediate, not long-term, child and family 
outcomes. Many states will need technical assistance to help them conduct evaluation in small, 
incremental steps, while developing a more comprehensive, long-term plan. 
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Introduction 

On July 1, 1993, Virginia implemented a major reorganization of services for youths with serious 
emotional and behavioral problems. Through the Comprehensive Services Act (CSA) for At-Risk Youth 
and Families, a system of services was to be created that was "child centered, family focused and 
community based." The CSA requires local interagency teams to plan and manage services and 
consolidated eight categorical funding streams to 
fund services. The new system of services is intended to: 

• increase interagency collaboration and family participation in service delivery and management; 

• dentify and intervene early with young children at risk and their families; 

• encourage public-private partnerships; 

• provide communities flexibility in the use of funds; and 

• reduce the use of more restrictive services, including out-of-home placements. 



A five-year evaluation plan was developed to be implemented in two stages. During the first stage, 
beginning July 1, 1993, the focus was on assessing the implementation of the CSA. During this stage of 
the study, the questions related to program implementation and interagency collaboration and funding 
were emphasized. Beginning June 1995, the research components related to assessing the impact of the 
projects on individual youths and families will be implemented. 

The focus of this report is on the implementation of the CSA in fiscal year 1994. 

Study Goal and Evaluation Questions 

The goal of this study was to examine those evaluation issues which were related to the implementation 
of the CSA. These issues included: 

Youths Served &shyp;The number and characteristics of the youths being served through the family 
assessment and planning teams (FAPTs) and/or receiving pool-funded services. 

Team Functioning and Interagency CollaborationSc shyp;The composition, coordination and 
collaboration of community planning and management teams (CPMTs) and family assessment and 
planning teams (FAPTs). 
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Family Participation&shyp;The involvement of family representatives and family members in the CSA 
process. 

Public-Private Partner ship &. shyp;What is the experience during the first year of closer working 
relationships between public and private agency staff? 

Goal Achievement&shyp;CSA participants' opinions on progress made toward reaching the goals of 
the CSA. 

Return to Top 



Results and Implications 

Youths Served 

About 8,000 young people in Virginia were served through CSA pool funds during the first year. A 
majority (approximately 60%) of CSA youths were White, male and between the ages 
of 13 and 17. Most (91%) of the youths were "mandated," meaning they were entitled to services 
through federal and/or state mandates. 

Of all CSA youth served the first year who were mandated, 57% were in private, residential care. About 
66% of all youth served through CSA pooled funds were placed outside their homes. These figures 
include all foster care children. About 83% of all youth received services provided by the private sector. 
This figure, too, includes all foster care children. Six percent of mandated youth received 
non-residential, public services. 

Team Functioning/Interagency Collaboration 

A statewide survey mailed in April 1994 to members of all CPMTs and a sample of FAPTs addressed 
issues related to team processes and development. About 1,100 surveys were distributed statewide. 
There were 450 CPMT respondents and 89 FAPT respondents to the survey. 

Information gathered from CPMT chairpersons revealed that there had been positive collaborative 
relationships between their teams and the local school system, local government, and local judges, 
though there is room for improvement in the latter case. 

In addition, FAPT members were more likely than CPMT representatives to report that their team 
functioned cohesively, with a common purpose. Moreover, FAPT respondents agreed at a lower rate 
than their CPMT counterparts that they had a sufficient number of CSA training opportunities, and that 
they had adequate opportunities to learn about other agencies represented on their teams. 

A special study was designed to explore interagency collaboration among the five agencies participating 
on the CPMTs. Findings of this research suggest that local agencies had already set the stage for 
effective collaboration prior to the inauguration of a "comprehensive" approach to services. One area 
that seemed to need attention&shyp;and apparently received it during the first year&shyp;was the level 
of agreement among agencies regarding the needs of at-risk youth and families, and the services 
necessary to meet these needs. 

Two key factors in the study were predictors of success in collaborative efforts. One was 
"respon-sibility"&shyp;the degree to which agency representatives believed that each collaborating 
agency carried out its responsibilities and commitments to other agencies. The other factor was 
"satisfaction"&shyp;the overall level of satisfaction members had with the other public agencies 
represented on their CPMTs.. 



All five agencies received high ratings on responsibility from at least 60% of their peers. Likewise, all 
agencies received high ratings on satisfaction from at least 56% of their peers. These findings and others 
point to an enhancement of interagency collaboration through the CSA team processes. Moreover, it is 
likely that agencies will engage in future collaborative efforts to address the needs of troubled youth and 
their families in Virginia. 



0 

ERIC 



27 



12/10/98 12:01 PM 



http://lumpy.fmhi.usf.edu/CFSroot/rtc/proceeding8th/8th.23.htmI 



8th.23.html 



Family Participation 

FAPT and CPMT members were surveyed about parent representatives on their teams. Responses were 
generally positive. 

• 87% of CPMT respondents and 82% of FAPT respondents agreed that meetings were scheduled at 
times parents could attend. 

• 79% of CPMT respondents and 88% of FAPT respondents agreed that parent representatives 
participated in team decision-making. 

• 74% of CPMT respondents and 78% of FAPT respondents agreed that parent representatives had a 
good understanding of the problems of at-risk children and youth and their families. 

• 65% of FAPT respondents agreed that participation of the parent representatives on FAPTs had 
resulted in improved service plans for at-risk youth and their families. 

• 59% agreed that services have improved with parent participation. 

• FAPT members were asked a set of questions about participation in the CSA process by parents of 
the youth served. Their responses were also generally positive. 

• 79% reported that parents attended FAPT meetings at which their children were discussed. 

• 82% said that the parents actively participated in these meetings. 

• 94% agreed that the presence of family members at FAPT meetings helped to ensure that the 
views of the youth's family were adequately represented. 

Public-Private Partnership 

CPMT survey respondents provided the following information about their perceptions of privately 
provided services. 

• 65% of CPMT respondents reported that privately provided services were available for local 
clients. 

• 64% said the availability of private services had not increased since the implementation of the 
CSA on July 1, 1993. 

• 70% thought that privately provided services were of "high quality." 

• 75% agreed that private agencies "demonstrated considerable knowledge" of the service needs of 
at-risk youth and their families. 

Finally, CPMT chairpersons were asked their perceptions of fees charged by private providers. Of the 53 
chairpersons who indicated having knowledge of private provider rates, 28 (53%) perceived an increase 
in fees since the beginning 
of the CSA. 

A majority of all survey respondents saw no increase in private services during the first year of the CSA. 
Such a change in the availability of services could not be expected to happen quickly, nor would it 
happen automatically, especially in areas which had no history in, nor infrastructure for providing 
adequate services to children. Nevertheless, the founders of the CSA hoped that private providers, 
working closely with public agency staff and parent representatives, would increase their services, 
especially those that were community-based. 

Goal Achievement 
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Among the seven major CSA goals, the one which received the most recognition of progress 
by both CPMT and FAPT survey respondents 

was interagency collaboration. There was also considerable acknowledgment of progress made in 
incorporating families into the service planning processes of the CSA. 

A majority of respondents perceived at least some or moderate progress toward the achievement of three 
goals: 



• provide communities flexibility in the use of funds; 

• provide services in the least restrictive environment; 

• improve the quality of services to troubled youths and their families. 

The two goals for which all respondents reported the least amount of progress achieved were 
public-private partnerships and early identification of, and intervention with, at-risk young children. It is 
understandable that more than 9-12 months would be needed to make substantial progress in these areas. 
In fact, it was surprising to observe the overall amount of goal achievement perceived by CSA 
participants during its first year of operation. 
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Conclusion 

In summary, survey respondents feel that the CSA has significantly changed the way in which Virginia 
addresses the emotional and behavioral problems of its young people. This report examined the 
successes and shortcomings of the CSA's first year of implementation. The focus 
was on the perceptions of those most directly responsible for the day-to-day operations of the CSA. The 
information they provided indicated that a tremendous amount of individual and group effort was 
expended to inaugurate the CSA. These efforts, moreover, led to significant accomplishments in a short 
period of time. Most notably, a high level of collaboration among agencies existed throughout most of 
the state. Through the CSA process, agency staff coordinated their efforts 

to deal with at-risk youths and their families. Moreover, survey respondents indicated that parents, both 
as community representatives and as guardians of troubled youth, have been meaningfully included in 
the process. 

Return to fop 



J. Randy Koch, Ph.D. 

Director of Research and Evaluation 

Albert C. Watts, Ph.D. 

CSA Evaluation Manager 

Audrey Jordan 

Research and Evaluation Associate 

Department of Mental Health, Mental Retardation and 

Substance Abuse Services 

P. O. Box 1797 

Richmond, VA 23214 

804/371-0802 Fax: 804/786-9248 

Return to Top 



O 

ERIC 



29 



12/10/98 12:01 PM 



8th .27 



http://lumpy.fmhi.usf.edu/CFSroot/rtc/proceeding8th/8th.27.html 



I Home | | search | | index | 

8th Annual Research & Training Center Conference Proceedings, Dept of Child and Family Studies, 
Florida Mental Health Institute, University of South Florida, 1996 

Access Vermont: A Statewide Evaluation of Non-categorical 
Service Delivery to Children and Families&shyp;Meeting 
Community Needs, Measuring Outcomes 

Authors 



Introduction Methods Discussion References 
Return to Table of Contents 



Introduction 

Access Vermont is part of the Comprehensive Community Mental Health Services Program for 
Children, a national services initiative funded through the Center for Mental Health Services (CMHS) 
under the Family Preservation Act to provide children's non-categorical mental health services in 22 
sites across the country. The grant will last for 5 years and began in October, 1993. Vermont is one of 
two states to receive statewide funding through this initiative. This paper will provide an overview of 
Access Vermont implementation and the evaluation component of Access Vermont with particular 
emphasis on: 

1 . the creation and implementation of an evaluation process that is individualized and which is 
beneficial to each local community; 

2. helping communities to incorporate evaluation into service provision; 

3. helping communities to empower children and families; and 

4. approaches to measuring outcomes of community-based, non-categorical services. 



Access Vermont: Implementation 

Background of Non-categorical Services in Vermont 

Over the past decade, Vermont has made considerable progress in developing a statewide, community 
based system of care for children and adolescents experiencing severe emotional and behavioral 
disturbances and their families. On the individual services level, Vermont developed a statewide 
capacity to provide community-based wraparound services with funding from Robert Wood Johnson 
(RWJ). On the systems level, funding from CASSP allowed each of the twelve local areas in Vermont to 
establish Local Interagency Teams (LITs) made up of parents, community members, and local agency 
representatives. 

Access Vermont has built upon these systems of care established under RWJ to help provide 
community-based, locally planned and implemented, non-categorical services. Through these systems of 
care, Vermont has been able to reduce the rates of out-of-school-district, out-of-home and out-of-state 
placements of children and youth, as well as the numbers of children taken into state custody (Vermont 
Department of Mental Health and Mental Retardation, 1993). 

Through the implementation of innovative programs and interagency teams, Vermont's mental health 
service system has been effective in serving many of the state's most challenging children and youth in 
their homes and communities, rather than in more restrictive treatment settings. 
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Long Term Goals of Access Vermont 

The long term goals of Access Vermont are to continue to reduce the rates at which Vermont's children 
are being removed from homes, schools and communities to more restrictive treatment settings, and to 
demonstrate better outcomes and reduce costs in the process. These goals will be accomplished through: 

1. increasing the quality of services available for children and families in home and school settings, 
with particular emphasis on effective and timely crisis response; 

2. evaluating and feeding back consumers' responses about their satisfaction with services and their 
empowerment within the system; and 

3. effectively building evaluation, with an eye towards outcomes, into service provision. 
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Access Vermont: Evaluation 

Development 

While the Local Interagency Teams have served primarily as case review teams in the past, the LITs are 
now moving towards policy making, by taking primary responsibility for local interagency planning, 
coordination and services integration. Through the LITs, community based services have been designed 
by local areas to meet their individual needs and to build on existing strengths. At this writing, 1 1 of the 
12 proposals for services have been reviewed and approved by the Vermont Department of Mental 
Health. Each catchment area receives money to implement these Local Priority service plans. These 
individualized services, inspired by community needs, are the heart of Access Vermont. 

Services that are being implemented under Access Vermont by local areas include crisis outreach and 
stabilization, crisis hotlines, short-term residential crisis stabilization programs, family crisis response 
teams, respite care, day treatment programs (school-based programs, after school and summer programs, 
day programs for transition aged youth), detoxification facilities, transition services, and a family 
support center /see Figure 1). 

Evaluation 

With all the different, individualized services being implemented and offered under Access Vermont, 
evaluation can be a challenge. Therefore, evaluation is being conducted on two levels&shyp;the 
individual level of the child and family, and the systems level of the service providers. 

Consumer Evaluation 

A vital part of Access Vermont is helping communities to empower children and families, and helping 
communities to evaluate empowerment and satisfaction. Families will be asked about their 
empowerment and satisfaction with prior and current services through measures (Family Empowerment 
Scale, Koren, DeChillo & Friesen, 1992; Client and Youth Satisfaction Questionnaires) administered 2 
weeks after the start of multi-agency services by members of the Evaluation Team (staff from the 
Vermont Department of Mental Health, the University of Vermont, and a Parent-Interviewer, hired 
through the Vermont Federation for Families). 

Demographic information and a number of standardized measures (CBCL: Achenbach, 1991; Youth 
Self Report: Achenbach, 1991; Child and Adolescent Functional Assessment Scale: Hodges, 1994; and 
Residential Living Environments Scale: Hawkins, Almeida, Fabry & Reitz, 1992) will be completed at 
intake by service providers for all consumers receiving services under Access Vermont. Follow-up 
interviews, consisting of the empowerment, satisfaction, and standardized measures will be conducted 
by the Evaluation Team with a random sample of consumers at 6 months, 1, 2, and 3 years after Intake 
for those consumers still receiving services. Youth and parents will be reimbursed individually for their 
efforts in these follow-up contacts. 
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Addressing the Evaluation Needs of Local Areas 

All of the measures listed above will be part of the evaluation at all 22 sites. However, the evaluation 
team is also working with each of the 12 local sites to evaluate any additional outcomes that the site may 
want to address. 

For example, one predominantly rural area that is implementing crisis outreach services wants to reach 
previously unserved, rural populations, and to assess how many "saves" they accomplish&shyp;that is, 
how many children and youth they successfully prevent from crises that would have result in 
out-of-home placements. Their evaluation centers around these issues, as well as evaluation of prior 
service involvement to assess how many previously unserved consumers they are reaching. 

Some areas will also be working closely with the schools to evaluate educational outcomes. The 
evaluation team is working with these areas to help them get the schools involved, and use outcome 
measures that will best serve their needs. 

System Evaluation 

Each LIT will serve as the hub connecting the local network of services used by children experiencing 
severe emotional disturbance. In order to evaluate how the different interagency systems are working 
together, LIT members will complete the Local Interagency Collaboration Checklist (Froelich, 1992) at 
the implementation of Access Vermont services, and annually thereafter for 3 years. Information 
obtained from this measure will be fed back to local areas. 

Feedback 

As both direct service providers and researchers know, service provision and evaluation have often been 
viewed as mutually exclusive and therefore conflicting interests. Evaluation outcomes are often not seen 
by service providers, and all too often, research findings are not applied to create positive organizational 
or service delivery changes. In the Access Vermont project, we are committed to a collaborative model 
where research, evaluation, and outcomes will blend with the innovative services that are being provided 
by local areas under this project, to maximize the benefits for both service providers and consumers. 

One of the most important steps in this collaborative process is the creation of a feedback loop, where 
service providers will receive timely periodic feedback on service outcomes and empowerment 
measures. Feedback will include not only standardized measures, but specific outcomes that individual 
communities have identified as important. 

We feel that one of the most effective ways to blend services and evaluation is to incorporate 
standardized assessment and outcome measures into intake procedures and clinical planning. By 
working collaboratively with service providers, assessment tools such as the CBCL can become 
beneficial in assessment procedures and ongoing clinical planning. A vital component of both local 
ownership and clinical relevance is helping service providers to utilize the standardized measures for 
assessment and treatment planning. The evaluation team will work to provide local areas with training in 
the clinical use of the assessment measures. We believe that empirical assessment measures will not only 
help to evaluate effective services, but can also aid in the provision of the most effective and relevant 
services. 



Quarterly data reports will also be sent to local areas to give summary feedback on all measures for the 
consumers they have served. These reports will help service providers gain a broader picture of the 
children and families they are serving, the types and hours of services they are delivering, and the 
difficulties and challenges of the consumers they are serving. Data will also be fed back in customized 
reports, in ways that local areas and agencies have deemed most useful to them. 



One exciting project the evaluation team has begun work on is the Access Vermont Newsletter, filled 
with items that are relevant to the project. We are encouraging LIT members, service providers, and 
anyone else involved with Access Vermont to contribute ideas or items to the newsletter. We are very 
excited about this project and hope that it will aid in feedback and communication between local areas 
and the evaluation team. 
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Conclusions 

Evaluation of non-categorical mental health services for children and families as innovative and as 
diverse as those planned under Access Vermont is certainly a challenging task. The evaluation itself has 
several aspects; it must support non-categorical children's mental health services, empower families and 
communities, and provide meaningful outcome data. In keeping with the system of care philosophy of 
family and community empowerment, both the services and the evaluation of those services are being 
designed by a collaboration among families, communities, mental health systems, and experienced 
researchers. 

It our belief that sharing our experiences in the design and implementation of this evaluation will be 
valuable, not only to the service systems and consumers of Vermont who are a part of Access, but to 
other communities, service providers and researchers who are implementing and/or evaluating similar 
services. 
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Introduction 

The Southern Consortium for Children (SCC; a regional mental health authority) and local county 
administrations have gradually changed the mental health service delivery system in Southeastern Ohio 
from a state-operated hospital model to a community-based, community-directed model. Program 
development has centered around the creation of children's mental health case management, therapeutic 
foster care, home-based services, and other services critical to providing a regional system of care. 
However, little information regarding the impact or effectiveness of system changes is available. With 
the emphasis on evaluation (Stroul & Friedman, 1994; both in terms of rigorous methodologies and 
naturalistic studies of program effectiveness) along with recent health care reform and the push for 
consumer driven service delivery (Burchard & Shaefer, 1992) the SCC increasingly interested in 
developing methods of ongoing evaluation. 

Conceptualization of Outcome 

This paper represents the initial efforts of the SCC to develop a system for ongoing evaluation of clinical 
outcomes for youth receiving services in collaboration with Ohio University. To address region specific 
needs, four areas of outcome assessment were considered: (1) problems associated with rural service 
provision; (2) perspectives of various stakeholders; (3) current research methods of outcome 
measurement; and (4) a theoretical and conceptual scheme of outcome assessment. 

Unique Problems in a Rural Setting 

Southeastern Ohio presents several unique problems for the provision of services and the development 
of an evaluation program. The service area is composed of ten counties located in the foothills of the 
Appalachian Mountains and in the heart of Ohio's 29 designated Appalachian counties. The terrain is 
rough and marked with poor roads and a non-existent public transportation system. The region is 
characterized by a sparse population, economic underdevelopment, and a low tax base. Not only do 
these difficulties impact the provision of services, but they necessarily affect the implementation of any 
outcome evaluation program. Evaluating service effectiveness must rely on brief, inexpensive methods 
of assessment that have intuitive value for the consumer. 

The Input of Stakeholders 

Strupp and Hadley (1977) proposed a tripartite model of mental health outcomes in which three 
interested parties consider the outcome of mental health interventions: society, the client (or consumers), 
and the mental health professional. Based on the viewpoint of the stakeholder (Gold, 1983), different 
criteria are selected to measure successful treatment. An initial step in establishing an evaluation plan is 
to make inquiries into these populations with regard to what kind of outcomes they deem are important. 
We used a Social Validation Questionnaire (VanDenBerg, Beck, & Pierce, 1992) to provide input from a 
variety of viewpoints regarding outcome assessment in 
our region. 
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Research Input 

Studies investigating the effectiveness of mental health services for children were also examined to aid 
the outcome development plan (e.g., Duchnowski, Johnson, Hall, Kutash, & Friedman, 1993). The 
review focused on instruments used to evaluate outcome and identified relevant domains of outcome. 
While all areas of assessment were not included, reviewing well-designed studies helped to ascertain the 
most important domains of assessment. 

Use of a Theoretical and Conceptual Measurement Classification Scheme 

Because of the divergent processes that occur during mental health intervention, divergent methods of 
measurement are used to capture the complexity of human functioning and change. Selecting from 
among these methods may be difficult if not guided by a theory. An ideal scheme may give purpose and 
direction to the selection of a final assessment package. The conceptual scheme used in this plan 
(Lambert, Ogles, & Masters, 1992) includes four theoretical dimensions upon which outcome 
instruments vary: (1) the content area addressed by the instrument, (2) the source of outcome ratings 
sampled by the instrument, (3) the outcome instrument's method or technology of data collection, (4) 
and the time orientation or stability of the instrument. The scheme was used to help select and develop 
an outcome assessment package that would tap a variety of content, source, technology, and stability 
dimensions. 
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Summary of Conceptualization 

Based on theses considerations, a list of desirable characteristics for measuring clinical outcomes was 
developed: 

1 . Outcome assessment must be pragmatic in terms of time and expense. Many research-based 
instruments are too cumbersome for ongoing clinical use. 

2. Outcome data must be understandable for paraprofessionals, parents and children. 

3. Effective outcome assessment should include input from multiple sources. 

4. Multiple content areas of outcome should be considered. 

5. Pragmatics should be counterbalanced by psychometric properties, including: reliability, validity, 
and sensitivity to change. 
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Development and Initial Evaluation of the Ohio Scales 

Based on our conceptualization, a set of brief, pragmatic outcome measures were developed. Three 
parallel forms of the Ohio Scales (OS) were developed for completion by the youth's parent or caretaker 
(OS-P), the youth (OS-Y), and the agency worker (OS-A). Four domains of outcome were selected: 
severity of problems, level of functioning, hopefulness, and satisfaction with treatment. Parents and 
youth rate all four areas. Case workers rate the youth using the problems and functioning scales. 

To begin evaluating the psychometric properties of the instrument, two samples of data were collected. 
For the community sample, approximately 297 Jr. High and High School students (average age 14.36, 
SD 1.54) completed the OS-Y. In addition, 281 parents (average age 39.43, SD 7.36) of each respective 
youth completed the OS-P. For the clinical sample, case managers at two agencies rated a total of 78 
youth (average age 14.28, SD 2.16). Additionally, 16 youths from this population completed the OS-Y 
and Youth Self Report (YSR; Achenbach & Edelbrock, 1983) and 28 parents completed the OS-P and 
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Child Behavior Checklist (CBCL; Achenbach & Edelbrock, 1983). Table 1 provides basic descriptive 
statistics for the OS problems, hope, and functioning domains for each respective population. 

Reliability 

Internal consistency data for each OS category is presented in Table 2. The internal consistencies for 
each category are adequate. Additional projects are underway to assess the inter-rater and test-retest 
reliability of the Ohio Scales. 

Validity 

Evidence of the concurrent, discriminant, and construct validity of the OS was also collected. Parent and 
youth ratings of problem severity and functioning were correlated with the CBCL and YSR (see Table 
3). The CBCL and YSR were included primarily to assess concurrent validity of problems. Additional 
studies are underway to validate the functioning scales. Case manager ratings of functioning and 
problems were correlated with the Progress Evaluation Scales (PES; Ihilevich & Glesser, 1979). 

The community sample provides some evidence for the discriminant validity of the OS. Five t-tests were 
conducted using parent ratings of functioning as the dependent variable to examine differences between 
students who had repeated a grade, been arrested, received mental health services, been assigned to 
classes for students with behavioral problems, or assigned to classes for students with learning problems, 
and those who had not experienced these events. Students who had been assigned to classes for youth 
with learning difficulties, had received previous mental health services, or had been arrested had 
significantly poorer functioning and more severe problems than students who had not experienced these 
events. Students who had previously been assigned to classes for youth with behavior problems had 
poorer functioning (but not more severe problems) than students who had not been assigned to these 
classes. 

The PES were administered in conjunction with the OS- A in the clinical sample on two 
separate occasions (approximately 4 months apart) to assess sensitivity to change. Change scores on the 
OS-A problems and functioning scales significantly correlated with the PES (r = .54, and .56, P < .001) 
and indicate a relatively high degree of agreement between the two instruments with regard to client 
change. 
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Conclusion 

While by no means completed, the results of the initial investigation of the psychometric properties of 
the OS are quite positive. The Ohio Scales have adequate internal consistency and preliminary evidence 
of concurrent and construct validity along with sensitivity to change. Additional projects are underway 
to further examine these properties. By conforming to the rather stringent conceptual outcome 
requirements, the developers hope that the final result will be pragmatically useful yet methodologically 
rigorous outcome measures that can be used for practical, ongoing evaluation of services for children 
and youth. 
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Introduction 

The Child and Adolescent Service System Program (CASSP; Day & Roberts, 1991; Stroul & Friedman, 
1986), which received initial federal funding in 1984, was established to assist states and communities in 
their efforts to improve child and adolescent service systems. CASSP has promoted the development of 
systems of care that are child-centered and family focused, with the needs of the child and family 
dictating the types and mix of services provided. 

In 1988, when the California State Department of Mental Health (Cal-DMH) received federal funding to 
implement the system of care principles developed by CASSP, a system of care program in Ventura 
county had been successfully in place for several years and the positive effects on children's mental 
health services had been demonstrated (Feltman & Essex, 1989; Jordan & Hernandez, 1990). Also in 
1988, California State Assembly Bill 377 (AB 377) had been passed to expand the Ventura program into 
three additional counties (Attkisson & Rosenblatt, 1993; Rosenblatt & Attkisson, 1993; Rosenblatt & 
Attkisson, 1992; Rosenblatt, Attkisson, & Fernandez, 1992; Ventura County Health Care Agency, 

1989). Therefore, the federal CASSP grant was seen by Cal-DMH as an opportunity to support and 
expand local system of care efforts while attempting to accomplish similar reforms at the state level. 

In 1993, an evaluation of the impact of CASSP on California's mental health service system for children 
and adolescents with serious emotional disturbance was conducted. Objectives included (1) evaluation 
of CASSP efforts to develop interagency and interdepartmental coordination and collaboration, (2) 
evaluation of CASSP efforts to develop family participation and advocacy related to services for 
children and adolescents with serious emotional disturbance (SED), and (3) evaluation of CASSP efforts 
to promulgate cultural competence in systems of care for children and adolescents. A comprehensive 
description of the procedures and results of this evaluation have been previously reported (Madsen, 
Dresser, Rosenblatt, & Attkisson, 1994); the current analysis will focus on the similarities and 
differences between informant groups (county human service agency staff and parents of children and 
adolescents with SED) in their reports of the impact of CASSP and the changes in California's service 
delivery system over the last four years. 
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Method 



To accomplish the evaluation objectives, three studies were conducted: (1) face-to-face interviews with 
county level agency staff, (2) telephone interviews with members of a statewide parent organization, and 
(3) a comprehensive review of relevant federal, state, county, and local documents. Selected results of 
the first two studies will be presented here, while results of the third study 
are presented in a companion paper (Dresser, Clausen, Rosenblatt, & Attkisson, 1995). 



Subjects 
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In the first study, of the 86 persons identified by state and county mental health staff as potential key 
informants in the selected 21 counties, interviews were conducted with 60 respondents (70%). Subjects 
for the second study were members of a statewide organization of parents and family members of 
children and adolescents with SED, the Family Network of California (FNC). Interviews were 
conducted with 67 respondents (31.5% of the identified sample) across 23 counties in California. 

Measurement 

A 45 -minute, semi-structured interview was developed for use with management and program staff in 
county departments of mental health, social services, juvenile justice, and education. Respondents were 
asked to estimate the impact of California CASSP in several areas, as well as to rate the changes in 
county service systems, using a Likert-type scale of 5 items, ranging from 1 (none or no impact) to 5 
(extensive impact). Informants were also asked to respond to several open-ended questions regarding 
changes in service systems for children and adolescents with SED over the last four years. 

The face-to-face semi-structured county interview format became a model for development of a 
semi-structured 30-minute telephone interview for use with parents of children and adolescents with 
SED. 
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Results and Implications 

Interagency Collaboration 

When asked to rate the impact CASSP had on increasing interagency collaboration in the delivery of 
services to children and adolescents with SED, both agency staff and parent informants indicated, on 
average, that CASSP had some impact in this area; however, the parent sample appeared divided on this 
issue, with many rating CASSP's impact as significant (25%) and many others rating it as minimal 
(17%). All of the county interviewees responded yes when asked if agencies in their county were 
working together in important ways; however, only 60% of the parents interviewed responded yes to this 
question. Overall, interview results suggest there is a perception that collaboration between agencies 
providing human services to youth with SED increased modestly throughout the State of California 
during the four years of CASSP implementation. 

Family Participation 

When asked to rate CASSP's impact on parent involvement in service development, implementation, and 
evaluation, county informants indicated a greater impact (mean = 3.36 on a scale of 1 - 5) than did parent 
informants (mean = 2.93 on a scale of 1 - 5), with most people in both groups (68% of county 
informants and 58% of parents) indicating CASSP had some (> 3 or some on a scale of 1-5) impact in 
this area. As compared to four years earlier, 82% of county agency informants and 58% of parent 
informants felt that service providers were more aware and supportive of parents and families of youth 
with SED. Thus, although subject responses to open-ended questions demonstrated that families of 
youth with serious emotional disturbance perceive human service agencies as inaccessible and 
frustratingly complex, there is evidence that CASSP had an impact on increasing the involvement of 
parents in human service planning and delivery. Analysis of comments made by county and parent 
interviewees suggests that CASSP's impact on increasing family participation was especially notable in 
the areas of staff and parent training, empowerment of individual parents, and organization of local and 
state parent groups. 



Cultural Competence 

Informants were asked to rate the extent to which their county valued cultural competence in service 
systems for children and adolescents with SED. It is in this area that county agency staff and parents of 
youth with SED most differed in their perception of county service systems. Indeed, almost two-thirds of 
county agency staff reported that their county placed significant or extensive value on moving systems 
of care toward becoming more culturally competent while two-thirds of parents reported that their 
county placed no, minimal, or some value on this issue. One possible interpretation of these findings is 
that, although service agency personnel have become increasingly aware of the importance of making 
service delivery culturally competent, changes have not yet been implemented at the direct service level 
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and have therefore not been perceived by service consumers and their families. Overall, results suggest 
that the perceptions of modest improvements in the cultural competence of service delivery were due in 
part to CASSP's efforts to inspire county mental health administrators to become leaders in the 
movement to establish culturally competent systems of care for children and adolescents. 
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System Change and Outcome 

When asked whether or not they believed systemic changes had occurred in their county's service 
delivery system for children and adolescents with SED, almost three-fourths of county agency staff and 
over half of parent informants responded yes. However, when asked to rate the extent to which the 
human service needs of children and adolescents with SED in their county were being addressed, over 
two-thirds of county agency staff informants and almost three-fourths of parent informants reported that 
those needs were addressed minimally or not at all. Several respondents noted that, although many of the 
needs of children and adolescents who have been identified are being addressed, there are many more 
youth with SED who are not yet identified by the system. 

In summary, the major findings regarding the perceptions of county service agency staff and parents of 
children and adolescents with SED about the impact of CASSP in California and changes in systems 
which serve youth with SED are as follows: 

1. a majority of both groups state that CASSP has had some impact on improving interagency 
collaboration between county human service agencies; 

2. both groups feel that human service agencies are working together in important ways, though 
human service agency staff are more in agreement with this statement; 

3. both groups feel that CASSP has had some impact on parent involvement in mental health service 
systems for children with SED; 

4. generally, both human service agency staff and parents of children with SED state that mental 
health service providers are more aware and supportive of parents and families of children with 
SED than they were four years ago; 

5. human service agency staff report the perception that agencies highly value cultural competence in 
service delivery while parents of children with SED report that agencies minimally or somewhat 
value this concept; 

6. both human service agency staff and parents of children with SED feel that there has been a 
systemic change in service delivery systems to children with SED over the last four years; and 

7. on average, human service agency staff feel that the needs of children with SED are being 
adequately met, while the majority of parents state that such needs are only minimally met. 
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Introduction 

The Child and Adolescent Service System Program (CASSP) in California is designed to compel system 
of care development for children and adolescents with serious emotional disturbance. As part of the 
evaluation of the CASSP in California (Madsen, Dresser, Rosenblatt & Attkisson, 1994), a review of 
documents was conducted to gain an understanding of the status of human service reform efforts in 
California. Documents reviewed include: (1) federal system of care policy and program descriptions; (2) 
state memoranda, documents, and reports relevant to CASSP implementation in California; (3) reports 
from county administrators involved with system of care development; and, (4) child advocacy and 
parent support group newsletters and monographs. The goal of this review of written products was to 
provide a broad context within which to assess 
CASSP contributions in California. 

Return to Top 



Method 

Documents were identified for review after consultation with state and county administrators, human 
service and research professionals, parents, and child advocates. Additional documents were also 
selected from the emerging literature on the design and evaluation of systems of care in California. 
Altogether, thirty-five documents representing over 7,500 pages of text were selected for review. The 
documents selected for review were organized into four broad types to enable reviewers to consider 
aspects of service system development from the perspectives of federal, state, county, and consumer 
participants in the reform process (See Table P. Reviewers developed detailed notes on the content of 
each document. From the notes, major themes, continuities, and discontinuites across documents were 
identified by project staff 

and synthesized into a summary of document contents. 
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Results 



The results section provides a summary of major themes and continuities regarding human service 
reform in California. While there was some variation in coverage and emphasis across documents (see 
Madsen, Dresser, Rosenblatt & Attkisson, 1994), the content of the documents relate to one of three 
themes: (1) challenges facing California; (2) factors that facilitate human service reform in California; 
and, (3) CASSP contributions in California. 



Challenges Facing California 

California faces many challenges to state-wide human service reform. Described below are a few of the 
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challenges discussed by document authors. 

1. Geographic and Population Scope and Diversity. California's population is nearly 32 million people. 
The state has the seventh largest economy in the world GY*. There is a diversity of geopolitical regions 
(urban, suburban, rural, mixed) and extraordinary ethnic diversity in many parts of the state. Population 
growth is driven in great part by immigration. Most immigrants to California are ethnically diverse 
people of child bearing age from Mexico, Central and South America, Pacific Rim, and Asian lands (4). 

Geographically large, densely populated, and ethnically diverse regions of the state with large numbers 
of low income residents (e.g., Alameda, Los Angeles, and the large southern counties of the state) pose 
special challenges to adequate service provision. Human service providers are challenged to become 
more knowledgeable and sensitive to the culturally specific needs of ethnically and economically diverse 
populations of children and families. Public schools in a number of California counties face complex 
problems stemming from large numbers of non-English language speaking children coupled with 
inadequate numbers of cross-culturally trained and bi- or multi-lingual educators (9). 

2. Economic Recession and Erosion of the Tax Base. The last national economic recession produced 
severe contractions in California's construction, computer, and defense industries and resulted in a 
decline in the average earnings of Californians. Hardest hit were the lowest 40 percent of wage earners, 
the unemployed, and the very poor who are most dependent on public services (3). 

The erosion of California's tax payer base provides less opportunity for local governments to keep pace 
with increased case load demands for public service. Voter and legislative initiatives, such as the 
Proposition 1 3 limitations on property taxation, the Proposition 98 set-aside of 40% of state funds for 
K-12 education, and the Gann limits on state spending, have severely curtailed the proportion of the state 
budget available for discretionary spending. Increased demand for public services is difficult to 
accommodate in California, where over 90% of the state budget is allocated to fixed costs and 
entitlement programs. 

3. Complex, Convoluted and Inflexible Funding. Health and human service administrators in California 
must navigate a complex array of federal, state, and local funding streams and private sector (insurance 
or foundation) support to marshal resources on behalf of children and youth. There are no less than eight 
federal funding streams relevant to mental health services alone, each of which finance very specific 
services to a very narrow target population of youth. 

A study conducted by an external consultant to the California State Department of Mental Health (2) 
revealed that California could significantly expand resources available for prevention efforts and 
community based services by lifting self-imposed limitations on access to and use of federal entitlement 
funding streams. Additionally, financial incentives are typically structured into the formulas for 
calculating in-home vs. out-of-home placement rates for each public human service sector in California 
&shyp; mental health, child welfare, juvenile justice and special education. Across 
the board, state and county cost sharing ratios produced incentives for counties to shift costs to the state 
through increased reliance on out-of-home placements (11). 

4. Lack Of Objectives and Accountability. Human service delivery systems throughout 

the state generally suffer from a lack of well-defined objectives, and a lack of accountability for 
documenting child, youth, and family outcomes. Without clear service objectives, and criteria for 
evaluating effectiveness, administrators and providers operate within a performance and information 
vacuum. 



Factors that Facilitate Human Service Reform in California Factors that facilitate human service reform 
in California include policy, fiscal, and programmatic changes that have been influenced by national 
trends, funding opportunities, state legislation, activist government, and parent leadership. Below are a 
few of these factors. 



1. Mental Health Program Realignment. Program Realignment (AB 948, the Bronzon-McCorquodale 
Act of 1991) is a legislative change in California with far reaching consequences for the financing. 
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organization and delivery of mental health services. Program Realignment was prompted by the state 
budget crisis (a $14 billion dollar state budget deficit in FY 1991-92) and constraints on the state budget 
process limiting discretionary expenditures for health and human services. The bill realigns fiscal 
responsibility by shifting costs, control, some revenue generation, and responsibility from the state to 
counties for public mental health, public health, some social services, and some state psychiatric hospital 
and locked facility beds. The cost shifting involves over $2 billion dollars worth of service provision 
annually (3). Realigned programs are funded through increased vehicle license fees and a permanent 
one-half cent sales tax increase. Formal research and evaluation of Program Realignment in California is 
not yet complete. 



However, an interim assessment (7) notes improvements in state financing of mental health services 
derived from Program Realignment. The improvements stem from: (a) a guaranteed revenue stream 
which enables county administrators to plan ahead and county governments to serve as more reliable 
contracting partners; (b) relaxed financing restrictions of state bureaucracy which enables counties to use 
de-categorized Medi-Cal and private funds more aggressively and carry over unexpended funds from 
one fiscal year to another; and, (c) increased county fiscal responsibility for service provision which 
shifts the incentive structure away from high rates of utilization of expensive institutional services. 

2. Managed Mental Health Care. The California Department of Health Services (DHS) managed care 
initiative (6) integrates California's two Medi-Cal funding streams for mental health services 
(Short-Doyle/Medi-Cal and Fee-For-Service/Medi-Cal) and places Medi-Cal mental health services 
under the domain of county mental health departments. Counties have a choice to serve as the Local 
Managed Mental Health Care Plan (LMMHCP) offering an array of services that emphasize prevention, 
promotion of health and functional ability, and decreased reliance on hospital and institutional services. 
In the managed care plans, individual providers are linked together under the umbrella of the LMMHCP. 
All eligible Medi-Cal beneficiaries receive their mental health services through the LMMHCP in 
accordance with access criteria and standards set by the state and Medicaid. 

3. System of Care Development. Legislatively enabled, state funded, and independently evaluated 
systems of mental health care are evolving in an expanding number of California counties (Rosenblatt, 
Attkisson & Fernandez, 1992; Rosenblatt & Attkisson, 1992, 1993; Attkisson & Rosenblatt, 1993; 
Attkisson, Dresser, & Rosenblatt, in press). System of care efforts were inspired 

and given definition by leaders from Ventura County in conjunction with local and state political 
representatives. Building upon CASSP principles, the Ventura leaders devised a planning model for 
reform of mental health service delivery (Feltman & Essex, 1989; Jordan & Hernandez, 1990). 

Following the planning model, Ventura successfully implemented community based mental health 
services for disturbed youth, demonstrated improved youth outcomes, and decreased overall 
expenditures. The initial 1984 demonstration in Ventura County (AB 3920) was expanded in 1987 to 
Riverside, San Mateo, and Santa Cruz (AB 377). More recently, through new legislation (AB 3015 
passed in 1993), the California Model System of Care was extended to an additional seven counties: 
Humboldt, Los Angeles, Merced, Monterey, Placer, San Luis Obisbo, and Stanislaus. San Francisco 
county pursued a system of care demonstration in the Family Mosaic Project with grant funds from the 
Robert Wood Johnson Mental Health Services Program for Youth (8). 



Collectively, system of care counties work toward desired outcomes for targeted populations of youth 
with serious emotional disturbance: controlled rates of out-of-home and state hospital placements and 
associated lower costs, improved school attendance and academic performance, 
and decreased juvenile justice recidivism. The system of care counties are implementing community 
based mental health services that involve improved financing for consumer choice and flexibility and 
individualization of service delivery. Parental, extended family, and foster family participation in 
program and treatment planning is an essential component of the California system of care movement. 
Other programmatic features include centralized and strategic case management to avoid unnecessary 
out-of-home placement, multi-agency coordination and treatment team approaches, and cultural 
competence of service delivery (10). The combination of demonstrating positive client outcomes while 
controlling public costs for services provided through multi-agency collaboration has facilitated 
tremendous support and momentum for systems of care in California. 

CASSP Contributions in California 
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Significant CASSP contributions in California include: 

1. Dissemination of System of Care Principles. Information dissemination regarding system of care 
concepts, along with funding opportunities for local system of care development, produced the 
knowledge and incentive systems necessary for the formation of a critical mass of system of care 
proponents throughout California. The CASSP provided organizing energy, technical assistance and 
formal trainings to county administrators, human service professionals, and parent groups seeking to 
improve local systems of care, enhance cultural competence of service delivery, and increase parent 
involvement in service systems for children and youth with serious emotional, behavioral and mental 
problems. Since CASSP development in California, system of care principles became an organizing 
theme for mental health service system development as reflected in the California State Mental Health 
Plans, the State Master Plan, county planning documents, system of care expansion and evaluation, and 
shifts in Medicaid funding policy. The system of care philosophy has been disseminated successfully 
among the California Mental Health Planning Council and the California Mental Health Directors 
Association, along with other key state-wide policy and planning groups. 

2. Formation of Broad-Based Advisory and Advocacy Groups. The CASSP supported the development 
of a state-wide multi-agency policy process through the role of the State 

CASSP coordinator on the State Interagency Advisory Council for Systems of Care (SIACSOC). 
Additionally, CASSP launched and supported the development of a formal family advocacy capacity in 
California by providing focused, ongoing assistance to parents involved with creating and growing the 
Family Network of California (FNC) and the United Advocates for Children of California (UACC) (5). 
CASSP also created a state-wide Cultural Competence Advisory Committee (CCAC) aimed at 
improving the cultural competence of service delivery to California youth and their families. 
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Conclusions and Implications 

In summary, despite significant challenges and obstacles to reform, a human service reform movement is 
underway in California; this movement has been inspired and supported by the national movement for 
integrated systems of care as exemplified by the Child and Adolescent Service System Program of the 
National Institute of Mental Health. The reform movement gained ground steadily in California during 
CASSP years, as evidenced by system of care expansion, mental health policy and financing changes, 
and increased leadership and activism on behalf of children, youth, and their families. Three thematic 
findings emerge from the review of documents: 

• There is a striking degree of consistency and agreement across sources about the identification of 
problems and level of progress achieved in reforming human services to meet the needs of 
California youth with serious emotional disturbance; 

• Human service reform is uneven across California counties, with state-funded, integrated system 
of care counties existing alongside counties apparently uninvolved in reform efforts; and 

• CASSP is a positive and constructive influence on human service philosophy, policy, and 
programs in California. Document authors emphasize CASSP contributions with respect to 
disseminating system of care principles and philosophy, and in fostering parent advocacy and 
multi-cultural involvement in mental health planning and policy making. State memoranda and 
reports document CASSP efforts and activities throughout California. County administrator 
reports and monographs reflect the positive role CASSP plays in local system of care development 
and enhancement. Parent advocates note the positive role CASSP serves in support of advocacy 
efforts. 
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Introduction and Program Description 

As the first study conducted to examine the clinical outcomes of a publicly funded system of care using 
a randomly assigned sample, the Stark County Evaluation Project encountered numerous 
methodological challenges. The purpose of this summary is to describe the process of implementing the 
study design, as well as to discuss the various decision-making mechanisms utilized and lessons learned. 



To date, there is no scientific evidence that the expanded care system is more clinically effective than 
traditional services. The recently completed Fort Bragg evaluation found that a continuum of care was 
not more effective than a traditional insurance system for military dependent children (Bickman, 
Guthrie, Foster, Summerfelt, Breda, & Heflinger, 1995). The primary aim of the research project was to 
assess the extent to which an expanded system of individualized care leads to improvements in the 
clinical functioning of children with severe emotional disturbance and their families. This 
research/demonstration was distinctive in that it examined the effects of change in an expanded system 
of care, in contrast to studying changes in individual components of the system (e.g., case management, 
intensive home treatment) and did not study a specially designed demonstration project but services 
provided without special funding. The site chosen for the study was Stark County, Ohio, a county which 
has several years experience of collaborative efforts in implementing a coordinated system of care. 

The Stark County project focused on two key questions: 

• What are the outcomes of an expanded system of care on the clinical functioning of children with 
severe emotional disturbance and on the functioning of their families? 

• What are the individual child and family factors that influence the outcomes? 

This summary of the Vanderbilt/Stark County Project will address process issues critical to 
implementing the research design. Outcome data will be published when the study is completed. 
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Method 



Criteria for Inclusion 

Participants were 360 children and adolescents (Stark County residents) who are between the ages of 5 
and 17.5 years inclusive, with indications of Severe Emotional Disorder (SED), and who were involved 
in at least one child-serving system (Department of Human Services (DHS), Education, or Juvenile 
Justice). Participants were recruited through several intake points with DHS and Child and Adolescent 
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Service Center (C&A), the child-serving community mental health agency. Children referred from DHS 
were screened by C&A and met intake criteria for C&A expanded services before a child was 
determined eligible for the study. 

Random Assignment 

A unique aspect of this study was the random assignment of children. Study participants were randomly 
assigned to receive the expanded services through C&A or standard care from community agencies 
outside of C&A. The random assignment procedure was felt to be the most valid scientific design that 
would enable us to determine if the expanded services were effective. Under this procedure, all 
participating children had an equal chance to receive the expanded services. Families were told about the 
two treatment conditions during the informed consent process, but they did not know to which condition 
their child was assigned until after the initial study interview. The study was designed to ensure that 
participants remained in the group to which they were assigned until data collection was completed. 

Data Collection 

The interviews with parents, foster parents, children and adolescents were conducted initially, and every 
six months for three waves of data collection. Participants received a combination 
of computerized, written, and, at follow-up, telephone interviews. The baseline data collection included 
self-administered computerized and written interviews. During follow-up interviews, all information was 
collected by phone or mail. 
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Lessons Learned 



Cooperative management was accomplished through a steering committee composed of representatives 
from Vanderbilt, Mental Health, and Department of Human Services. Monthly meetings were held to 
identify problems, air concerns, sort out turf issues, and reach decisions on how to address concerns. 
Good communication and the ability to remain flexible were key elements in project management. 
Identification of roles for each organization was crucial for implementation. Development of procedural 
protocols was also important. 

Vanderbilt played the role of providing data regarding recruitment, documenting services provided to 
clients, and encouraging the maintenance of standards required by the research design. Mental Health 
and Department of Human Services used the monthly meetings to continually coordinate their efforts at 
identifying untapped pools of potential participants, and to develop procedures for accessing those 
potential participants for the study. 

During implementation, barriers to success of the project were identified and addressed by the steering 
committee. The major barriers pertained to ethical concerns and participant recruitment. 



Ethical Concerns 

In implementing the research project with random assignment, several ethical questions had to be 
addressed. Those questions were raised by direct care providers in an effort to assure themselves that 
they were in compliance with their professional ethical standards. The ethical concerns raised were as 
follows: 



1 . Is it ethical to deny the control group access to a system of care that providers believe to be the 
best way of providing service to families? 

2. In such a study, is it ethical and/or practical to ask C&A therapists and case managers to provide 
limited service to control group subjects? That is, should the project attempt to treat the control 
group within the system but withhold the expanded services? 




3. Should experimental group clients be required to see only system of care therapists for services? Is 
it possible that a case manager could coordinate the service plan with a private therapist or 
provider from another agency? 
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4. If a control group client decided that they want to seek system of care services, can we deny 
access to those services? 



The steering committee used the Ethical Principles of Psychologists and Code of Conduct (APA, 1992) 
as their guide in addressing these questions. 

In addressing the first ethical concern, the steering committee debated the issue of professional opinion 
versus existing research findings. In his role as principle investigator, the Director of the project 
provided information substantiating the fact that no experimental studies have been conducted on this 
matter. It was the opinion of the steering committee that these were indeed professional opinions and not 
matters of fact. Therefore, there was no ethical bind for providers of service to participate in the study 
with knowledge that control group would not have access to the system of care. 

In regard to the second question, it was decided by the steering committee that it was not practical to ask 
system of care (experimental group) therapists to provide services to both the experimental group and 
control group subjects. Moreover, based on ethical issues, it was important to C&A that control group 
participants not be clients in their treatment system. By serving them outside of C&A, they were not in 
the role of denying services. 

Regarding question number three, it was decided that it was possible for a case manager from an 
experimental group participant to coordinate services with providers from any agency. This is indeed the 
work they do on a regular basis. Therefore, it was not considered to be an undue burden or an ethical 
concern for case managers to work with system of care therapists and/or therapists from other agencies 
or private practice. 

Regarding question number four, it was decided that if a control group subject decided to seek services 
from the system of care that they are in effect, dropping out of the study. However, detailed procedures 
were developed to assist those clients in finding other satisfactory providers that are not part of the 
system of care. By providing this assistance to clients who were not satisfied with their current provider, 
it allowed the study to maintain subjects in a more true control group setting as opposed to mixing part 
of their treatment with system of care therapists. It would have been clearly unethical to simply deny the 
client's request for the sake of the study. Once the client had dropped out however, they may request any 
service without restriction. 



Subject Recruitment 

Recruitment of subjects emerged as a difficult issue in the study. Mental health did not have the volume 
of clients to supply 360 subjects within the specified time frame of the research project. Department of 
Human Services (DHS) had the volume of clients, but lacked the clinical expertise to judge 
appropriateness of referrals for the study. Juvenile Court had problems similar to DHS. Schools had the 
volume, but were hesitant to engage in experimental research because of potential public relations 
problems. 



In the end it required a collaborative effort between Mental Health and Department of Human Services 
with much encouragement from Vanderbilt to recruit enough participants for the study. That 
collaborative effort consisted of Mental Health providing consultation in the form of screening potential 
referrals from DHS before the referral was actually made to Vanderbilt. This assured appropriateness of 
participants referred for the study and increased the volume to a level that would allow us to reach 360 
subjects within the time frame of the study. Other procedures were developed to recruit from juvenile 
justice as well as from local school systems. These efforts, however, were largely unsuccessful. 



Finally, procedures were developed for recruiting from area United Way agencies. These efforts were 
initiated by Mental Health because of their existing relationship with United Way. After initiating the 
dialogue, the Vanderbilt staff then followed up and designed procedures that would allow them to 
receive referrals directly from United Way agencies. Because of the amount of time involved in working 
out such collaborative procedures, it placed significant pressure upon Vanderbilt as the primary 
investigator in this project to maintain the study within the time frame specified with the National 
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Institute of Mental Health. 
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Summary 

Collaboration between Vanderbilt, Mental Health and the Department of Human Services was crucial in 
addressing methodological and procedural issues to maintain the experimental design. Future 
community-based research efforts would be well advised to establish a steering committee to assure 
good communication and successful implementation of research design. 

The Vanderbilt/Stark County Evaluation Project is the first study to examine clinical outcomes in a 
publicly funded system of care using random assignment of subjects. The study has two key questions. 
One was to evaluate what the outcomes of a systemic approach to care are regarding clinical functioning 
of children with SED and their families. A second question was to look at what individual and family 
factors influence outcomes. 
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Background 

For the past decade, the pendulum for children and adolescent mental health services in Region II in 
Louisiana has been swinging from traditional institutional care to services that are child-centered, family 
and community-based. As the pendulum has swung, a continuum of services has developed that allows 
families of children and adolescents with severe emotional disturbances to choose from an array of 
services ranging from traditional inpatient care to home-based services. A goal of the developing system 
is to become a "one-stop" and "user friendly" system of care. This paper describes our system of care 
and all of its service options. Further, this paper describes a plan that is being developed to collect data 
from the system so as to objectively and scientifically study the effectiveness of the system as a whole 
and its individual parts. 

Region II of the Office of Mental Health in Louisiana serves seven parishes. The Margaret Dumas 
Mental Health Center (MDMHC) serves children ages 6 to 18 in Baton Rouge and the surrounding 
parishes. East Baton Rouge Parish is the largest urban area in the region with the outlying parishes being 
mainly rural. There are over 8000 children with severe emotional disturbances under the age of 18 in this 
seven parish area. This figure is based on an estimated five percent of the population. Many of the 
children in the area are served through the psychosocial rehabilitation option plan funded by Medicaid. 
Those served through the MDMHC have access to the continuum of care discussed below. The majority 
of cases at MDMHC have been of the lower socio-economic status and uninsured. 

Return to Top 

The System of Care 

On a continuum from least restrictive to most restrictive, our current system of care includes family 
support groups, educational groups, traditional outpatient counseling, home and school-based services, 
case management services, respite services, home companions, school companions, intensive 
home-based crisis counseling, runaway watch, suicide watch, therapeutic foster care, day treatment and 
inpatient treatment. A child or adolescent may move in either direction on this continuum at any time 
during his/her treatment. Each treatment plan is tailored to meet the individual's and family's specific 
needs. The following are descriptions of each treatment option. 

Family Support Groups: A local chapter of the Louisiana Federation of Families for Children's Mental 
Health is operated through the Margaret Dumas Mental Health Center and meets regularly to provide 
support for families of emotionally disturbed children/adolescents. They also maintain a resource library 
which contains information pertaining to mental illnesses and other relevant topics. 

Educational Groups: Currently providing three educational groups: Attention-deficit Hyperactivity 
Disorder, Anger Management, and Parenting Skills. Open to children and families receiving mental 
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health services. 

Traditional Outpatient Services: Traditional counseling and medication services at the mental health 
center by appointment or in emergency situations. Three social workers, one psychologist, and three 
part-time psychiatrists provide services. 

Home and School-Based Services: Traditional counseling services, but within a more natural setting, 
such as the child/adolescent's home and school. 

Case Management: Assistance with a variety of physical, social, and recreational needs of the child and 
family; assistance includes coordination of services received and an advocacy role. 

Brief in-home or out-of-home services: Supervision of the child/adolescent is provided by 
paraprofessionals. Allows primary caregivers to have time off from supervisory responsibilities. 

Includes both planned and emergency respite. 

Home Companion: A paraprofessional maintains one-on-one contact with the child/adolescent during a 
specified period of time within the home. Assists with supervisory duties to relieve primary caregivers. 

School Companion: A paraprofessional maintains one-on-one contact with the child/adolescent during a 
specified period of time within the school. Assists teacher in keeping the child/adolescent on task and in 
maintaining self-control. 

Intensive Home-Based Crisis Counseling: A four to six week program of intensive home-based 
counseling to prevent out-of-home placement of the child/adolescent. The child/adolescent must be at 
moderate risk for out-of-home placement within the next few months to qualify. 

Runaway Watch: 24 hour, one-on-one coverage for the child/adolescent who is a significant threat to 
runaway from home. Allows primary caregivers to sleep at night, etc. Attempts to prevent out-of-home 
placement of the child, and is an alternative to hospitalization. 

Suicide Watch: 24 hour, one-on-one coverage for the child/adolescent who is a significant threat to 
commit suicide. Allows primary caregivers to sleep at night, etc. Attempts to prevent out-of-home 
placement of child and is an alternative to hospitalization. 

Therapeutic Foster Care: Temporary placement of the child/adolescent with a foster family for 
up to four months while problems with family or environment are resolved. An alternative to 
hospitalization. 

Day Treatment: Mental Health Services provided on a daily basis within a structured setting. However, 
the child/adolescent returns home each night. An alternative to hospitalization. 

Inpatient Treatment: Hospitalization for intensive inpatient services. Most restrictive of all options. 
Return to Top 



Barriers 



While the system of care provides many treatment options, it is limited in its resources. Inherent in the 
task of developing a system of care for the children in Region II is the need to create a stable, long-term 
financing mechanism to support both the development and the operation of such systems. Presently in 
Louisiana, there is not a special children's mental health budget. All funds over annually budgeted means 
of financing generated by either adult mental health or children's mental health services are returned to 
the State General Fund. This mechanism not only handicaps the day-to-day operation of the mental 
health system but also limits the possibilities of pursuing Federal Grants. On many occasions, the State 
was not able to meet the grant requirement of coming up with "matching funds." A creative way of 
generating and managing a special fund for the system of care is needed. Due to these funding 
restrictions, resources have to be monitored carefully. Therefore, it is imperative that the components of 
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the system be evaluated so that the most effective components are available to the most people in need. 
We are in the process of developing a program to evaluate the effectiveness of the system. 

In addition to the barriers presented by funding problems, there are also transportation problems for the 
clients and a need for more clinical and clerical staff. Other aspects of the program that we would like to 
see added to the continuum of care include an after school program and more recreational services. 

Data being monitored include broad demographic characteristics to allow studies on urban vs. rural 
factors, age, racial, gender, and economic factors, and individual school factors. Other data collected 
include diagnostic information, medications, service utilization (least restrictive to most restrictive), 
hospitalizations, and other treatment factors. These data will allow us to formally evaluate effectiveness 
and to perform studies aimed at deriving predictor variables which will help plan our system more 
efficiently. If we can determine specific characteristics of clients that predict success or failure within a 
specific component of the system, we will be able to more efficiently use the limited resources we have. 
Return to Top 
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